Sensor,  User,  Mission  (SUM)  Resource  Management 
and  Their  Interaction  with  Level  213  Fusion 


Erik  Blasch 

Air  Force  Research  Lab,  2241  Avionics  Cir 
WPAFB,  OH  45433,  erik.blasch@wpafb.af.mil 


Abstract  -  Revisions  to  the  JDL  model  by  the  current 
DFIG  team  (Data  Fusion  Information  Group)  include 
definitions  for  model  usefulness  that  stressed  various 
control  functions  of  sensor,  user,  and  mission  (SUM) 
management.  The  purpose  of  the  paper  is  to  highlight 
issues  and  challenges  to  real  world  separation  of  control 
actions.  This  position  paper  highlights: 

1 .  Addressing  the  user  in  system  management  /  control, 

2.  Determining  a  standard  set  of  metrics  for  optimization, 

3.  Evaluating  fusion  systems  to  deliver  timely  info  needs, 

4.  Dynamic  updating  for  planning  mission  time-horizons, 

5.  Designing  interfaces  to  support  impact  assessments 

Keywords:  Fusion,  Situational/Impact  Assessment, 
Resource/Sensor  Management,  User  Refinement 


1  Introduction 

In  the  companion  paper,  we  listed  many  of  the  issues  and 
challenges  in  Situation  Assessment  (SA  -  Level  2)  [1]  for 
knowledge  representation  and  reasoning.  In  this  paper, 
the  goal  is  not  to  reiterate  many  of  the  issues  and 
challenges  of  SA,  but  to  build  on  these  as  related  to 
specifically  Level  3  (Impact  Assessment)  and  Level  4 
(Process  Refinement). 

Hall  and  Llinas  [2]  address  various  surveillance  volumes 
and  for  sensor  platforms  (e.g.  Battlefield  Intelligence ,  10s 
to  100s  of  miles,  aircraft).  In  Level  4,  they  list  the  key 
techniques  of  Measurement  of  evaluation.  Measures  of 
performance,  and  Utility  theory.  The  issues  and 
challenges  listed  for  Level  4  processing  from  1997  are: 


Current  Status 

Challenges/  Limitations 

Robust  single  sensor 
Operations  research 
formulation 

Limited  approximate 
reasoning  app. 
MOP  /  MOE  focus 

Incoiporation  of  mission  objectives 
Environmental  context  for  sensor 

utilization 

Conflicting  objectives 

Dynamic  algorithm  selection 

Diverse  sensors 

The  key  challenges  expressed  were  (1)  limited 
communications  bandwidth  for  data  aggregation,  (2) 
context-based  approximate  reasoning  for  Level  3 
understanding,  and  (3)  knowledge  representation  for  Level 
2  processing.  While  many  of  these  issues  are  consistent 
with  the  ISIF05  panel  discussion  for  SA  processing  [1]; 
the  key  metrics  for  evaluation  have  not  been  adopted  by 
the  community.  Since  1997,  more  timely  and  localized 
information  is  needed  for  individuals  carrying  a  hand-held 


video  cell  phones.  The  interplay  for  Level  4,  Resource 
Management  (RM)  and  the  various  fusion  process  levels 
are  still  evolving  as  more  data  is  available.  [3] 

In  this  paper,  we  address  briefly  the  issues  associated 
with  resource  management  for  Level  2/3  (SA/IA) 
interaction.  [4]  As  stated  above,  while  SA  and  lA  are  less 
well  researched,  even  further  removed  is  the  S  A  -RM  and 
lA-RM  interdependencies.  Developments  for  tracking  and 
control  (Level  1/4)  have  been  addressed  [5^],  and 
communication  issues.  [7,8]  Also,  utility  and  risk 
assessments  P]  have  been  posed  for  SA/IA  interactions 
that  could  be  used  in  an  objective  function.  What  is 
missing  from  previous  research  is  that  the  Users  drive  the 
control  process,  from  the  inception  of  sensor  designs  to 
fusion  algorithm  approval. 

The  rest  of  the  paper  will  focus  on  issues  and 
challenges  for  SUM  management  as  related  to  level  2/3 
processing.  Section  2  explains  the  DFIG  model.  Section  3 
discusses  level  2/3/4  assessments  and  Section  4  discusses 
issues  for  resource  management. 


2  DFIG  Fusion  Model 

To  set  the  stage,  we  again  show  the  DFIG  I  (Data 
Fusion  Information  Group)  model  (as  the  upgrade  to  the 
IDL  [10,  11]  model),  shown  in  Figure  1.  In  this  model,  the 
goal  was  to  separate  the  information  fusion  and 
management  functions. 
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Figure  1 .  DFIG  2004  model. 
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The  key  in  the  DFIG  model  is  to  realize  that  differing 
control  functions  are  based  on  the  spatial  /  temporal  / 
spectrum  difference.  The  spectral  needs  are  based  on  the 
type  of  sensor  needed.  The  temporal  needs  are  based  on 
the  user’s  need  for  timely  information  to  afford  action. 
Finally,  the  spatial  needs  are  based  on  the  mission  goals. 
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Management  functions  are  divided  into  sensor  control, 
platform  placement,  and  user  selection  to  n®et  mission 
objectives.  SA  includes  tacit  functions  which  are  inferred 
from  level  1  explicit  representations  of  object  assessment. 
Since  the  unobserved  aspects  of  the  SA  problem  can  not 
be  processed  by  a  computer,  user  knowledge  and 
reasoning  is  necessary.  Level  3  (lA)  sense-making  of 
impacts  (threats,  course  of  actions,  game -theoretic 
decisions,  etc)  helps  refine  the  SA  estimation  and 
information  needs.  The  cun'ent  DFIG  definitions  include: 

Level  0  —  Data  Assessment:  estimation  and  prediction  of 
signal/object  observable  states  on  the  basis  of  pixel/signal  level 
data  association  (e.g.  information  systems  collections); 

Level  1  -  Object  Assessment:  estimation  and  prediction  of  entity 
states  on  the  basis  of  data  association,  continuous  state 
estimation  and  discrete  state  estimation  (e.g.  data  processing); 
Level  2  -  Situation  Assessment:  estimation  and  prediction  of 
relations  among  entities,  to  include  force  structure  and  force 
relations,  communications,  etc.  (e.g.  information  processing); 
Level  3  -  Impact  Assessment:  estimation  and  prediction  of  effects 
on  situations  of  planned  or  estimated  actions  by  the  participants; 
to  include  interactions  between  action  plans  of  multiple  players 
(e.g.  assessing  threat  actions  to  planned  actions  and  mission 
requirements,  performance  evaluation); 

Level  4  -  Process  Refinement  (an  element  of  Resource 
Management):  adaptive  data  acquisition  and  processing  to 
support  sensing  objectives  (e.g.  sensor  management  and 
information  dissemination). 

Level  5  -  User  Refinement  (an  element  of  Knowledge 

Management):  adaptive  determination  of  who  queries 

information  and  who  has  access  to  information  (e.g.  information 
operations)  and  adaptive  data  retrieved  and  displayed  to  support 
cognitive  decision  making  and  actions  (e.g.  human  computer 
interface). 

Level  6  —  Mission  Management  (an  element  of  Platform 
Management):  adaptive  determination  of  spatial -temporal 
control  of  assets  (e.g.  airspace  operations)  and  route  planning 
and  goal  determination  to  support  team  decision  making  and 
actions  (e.g.  theater  operations)  over  social,  economic,  and 
political  constraints. 

Issues 

1)  Level  2/3  tradeoffs  in  information  quantity  {throughput) 

2)  Timeliness  of  level  3  process  refinement  to  control 

sensing  needs  (e.g.  parameters  to  sense) 

3)  Level  3  use  of  domain  knowledge  to  predict  future 

sensing  needs  (i.e.  context) 

4)  Multiple  users  have  differing  levels  of  process  needs  in 

a  distributed  fashion  from  the  same  situation. 

5)  Varying  fidelity  of  confidence  reporting  of  impending 

threats  and  situations 

Challenges 

1)  Pedigree  analysis  to  backtrack  through  associations  to 

capture  the  impending  threat 

2)  Time  Horizons  of  control  actions  from  lA  to  update  the 

SA  (i.e.  priority  schemes) 

3)  Performance  models  of  level  1  analysis  to  afford  level 

2/3  information  needs  satisfaction  and  sensor 

management 

4)  Hierarchical  cost  functions  that  include  risk  and  utility 

analysis  of  level  4  processes. 

5)  Unified  set  of  metrics  that  afford  level  2/3  processing 

that  can  be  optimized  in  a  level  4  objective  function 


3  Level  2-6  Functions 

Level  2/3  Situation  assessment  /  impact  assessment) 
issues  resolve  around  a  bottom  up  or  top-down  analysis. 
SA  desires  as  much  local/  global  information  (bottom-up) 
[12]  to  increase  spatial  and  temporal  coverage  [13],  while 
lA  is  interested  in  dimensionality  reduction  (top-down)  for 
action.  lA  is  concerned  with  the  retaining  salient 
information  through  filtering.  [9,  14]  I A  filters  pertinent 
past  information,  estimates  and  prioritizes  intent  [15],  or 
predicts  consequences.  An  example  is  a  risk  analysis, 
which  uses  utility  theory  to  determine  the  potential  impact 
of  a  situational  state.  Another  analogy  might  be  the  breath 
versus  depth  analysis.  SA  is  searching  for  information 
{breath)  while  lA  is  linking  together  for  a  time  -chaining  of 
events  {depth).  For  varying  levels  of  threats,  there  are:  (1) 
Mission  Area,  (2)  Awareness,  and  (3)  Individual 
Awareness  Boundaries. 

Level  4,  Process  Refinement,  includes  sensor 
management  control  of  sensors  and  information.  In  order 
to  utilize  sensors  effectively,  the  IF  system  must  explore 
service  priorities,  search  methods  (breadth  or  depth),  and 
determine  the  scheduling  and  monitoring  of  tasks. 
Scheduling  is  a  control  function  that  relies  on  the 
aggregated  state-position,  state-identity  and  uncertainty 
information  for  knowledge  reasoning.  Typical  methods 
used  are  (1)  objective  cost  function  for  optimization,  (2) 
dynamic  programming  (such  as  NN  methods  and 
reinforcement-learning  based  on  a  goal),  (3)  greedy 
information-theoretic  approaches  p],  or  (4)  Bayes  net 
which  aggregates  probabilities.  Whichever  method  is 
used,  the  main  idea  is  to  reduce  uncertainty,  detect  events, 
and  afford  mission  planning.  With  many  possibilities  for 
PR,  the  user  must  agree  to  the  strategy  embedded  in  the 
sensor  management  control  function. 

Level  5,  User  Refinement,  functions  include  the  (1) 
selection  of  models,  techniques,  and  data,  (2)  determining 
the  metrics  for  decision  making,  and  (3)  performing  higher- 
level  reasoning  over  the  information  based  on  the  user’s 
needs.  The  user’s  goal  is  to  perform  a  task  or  a  mission. 
The  user  has  preconceived  expectations  and  utilizes  the 
level  04  capabilities  of  a  machine  to  aggregate  data  for 
decision  making  and  mission  completion.  For  example,  the 
user  plans  ahead  (forward  reasoning)  while  the  machine  is 
reasoning  over  collected  data  (backward  reasoning).  If  a 
delay  exists  in  the  IF  reasoning,  a  user  might  deem  it 
useless  for  planning. 

Level  6,  Mission  Management  functions  include  the  (1) 
addressing  the  cost  and  maintenance  cf  platforms,  (2) 
determining  where  to  place  assets,  and  (3)  coordination  of 
the  operators  of  the  sensors  and  platforms.  The  User  is  a 
team  that  responds  to  some  need  that  plans  a  mission 
(sensor,  platforms,  time-on-station,  look  angles,  etc).  The 
mission  determines  the  team,  which  instantiates  the  actors 
(i.e.  users)  as  a  subset  of  the  group,  from  which  platforms 
with  sensors  are  fueled  with  communication  bandwidth 
allocations,  and  set  in  motion  for  predictive  sensing. 


4  SUM  Issues 

4.1  User  Control 

The  key  for  RM  is  to  understand  how  the  user  reasons  for 
action  based  on  IF  results  including  trust,  workload,  and 
attention  [4]  and  perceptual  needs.  [16].  The  roles  that 
the  user  can  play  include  planning,  organizing, 
coordinating,  decision  andaction.  Kokar[17]  and  others 
[18]  stress  ontological  and  linguistic  questions 
concerning  the  user  interaction  such  as  semantics, 
syntatics,  efficacy,  and  spatio  -temporal  queries. 

4.2  Standard  Set  of  Metrics  for  Evalnation 

Designing  complex  and  often-distributed  decision 
support  systems  (data  — >  information  — >  decisions  — > 
plans  — »  actions)  requires  an  understanding  of  both  the 
fusion  and  user  processes.  DM  requires:  (1)  SA/IA,  (2) 
dynamic  responsiveness  to  changing  conditions,  and  (3) 
continual  evaluation  to  meet  information  needs.  [19] 
Important  aspects  of  fusion  include  timeliness,  mitigation 
of  uncertainty,  and  output  quality  relative  to  contexts, 
requirements,  and  constraints.  Standardized  absolute  and 
relative  assessment  metrics  for  evaluating  the  success  of 
deployed  and  proposed  systems  must  map  to  these 
constraints  for  reliability/integrity  through  verification  and 
validation  tests.  Many  IF  strategies  must  be  rigorously 
evaluated  by  a  standardized  method  over  various 
locations,  changing  targets,  differing  sensors,  and  IF 
algorithms.  [2]  Issues  must  involve  as  well  as  local  and 
global  metric  optimization  in  RM  including:  [20] 


Table  2:  Metrics  for  various  Disciplines. 


COMM 

User 

Info  Fusion 

4TR/ID 

TRACK 

Delay 

Reaction 

Time 

Timeliness 

Run  Time 

Update  Rate 

Prob,  Error 

Confidence 

Confidence 

ROC 

Prob.  Detect 

Delay  Var, 

Attention 

Accuracy 

Pos. 

Accuracy 

Covariance 

Throughput 

Workload 

Throughput 

No.  Images 

No.  Targets 

Security 

Trust 

Reliability 

Authorize 

Cooperative 

Cost 

Cost 

Cost 

#  platform 

No.  Assets 

4,3  Timely  Information 

Fusion  evaluation  requires  off-line  and  on-line 
assessment  of  individual  sensor  and  integrated  sensor 
exploitation  (i.e.  performance  models)  so  as  to  plan  a 
mission  and  react  in  a  timely  manner.  M  ission  management 
requires  putting  platforms  with  sensors  on  station  at  the 
correct  time,  preparing  operators,  and  determining  pending 
actions  with  estimated  data. 

When  tasked  with  an  SA/IA  analysis,  a  user  can  respond 
by  one  of  three  manners,  broadly:  reactive,  proactive  or 
preventive.  In  &  Reactive  mode,  the  user  makes  a  rapid 
detection  and  minimizes  damage  to  immediate  threats.  The 
individual  user  selects  the  immediate  appropriate  response 
(in  seconds)  with  aid  of  sensor  warnings  of  threats.  In  the 
Proactive  mode,  [21]  the  user  utilizes  sensor  data  to 
anticipate,  detect,  and  capture  needed  information  prior  to 


an  event.  A  Multi-INT  sensor  system  could  detect  and 
interpret  anomalous  behavior  and  alert  an  operator  to 
anticipated  threats  in  minutes.  The  mode  that  captures 
the  entire  force  over  a  period  of  time  (i.e.  an  hour)  is  the 
Preventive  Mode.  The  preventative  mode  includes  an 
Intel  database  to  track  events  hours  before  they  reach 
deployment  for  mission  management. 

4,4 Distributed,  Dynamic  Updates  for  Planning 

Intelligent  decision  making  requires  distributed, 
dynamic  and  timely  updates.  Standard  models,  such  as  the 
Observe-orient-decide-act  (ODD A),  estimate  individual 
user’s  planned,  estimated,  or  predicted  actions.  Assessing 
susceptibilities  and  vulnerabilities  to  detected,  estimated, 
and  predicted  adversarial  threat  actions,  in  the  context  of 
planned  courses  of  actions,  requires  a  concurrent 
timeliness  assessment  and  its  affects  on  team  decision 
making.  Such  assessment  is  required  for  adequate 
distributed  actors,  yet  is  not  easily  attained. 

Coordinated  complementary  and  orthogonal  actions  are 
needed  over  differing  timelines  and  geographical  areas. 
Mission  management  needs  address  global  control 
functions  that  helps  to  determine  which  sensors  and  users 
are  activated  in  for  local  control  responsibilities  and 
action  approval. 

4,5 User  Display  for  Interaction 

Fusion  models  process  data  for  user  decision  making.  The 
top-down  approach  explores  an  information  needs  pull  by 
queries  to  the  IF  system  Cognitive  fusion),  while  a 
bottom-up  approach  pushes  data  combinations  to  the  user 
(display  fusion).  The  main  issues  for  user-fusion 
interaction  is  the  query  ontology,  understandable  metrics, 
and  uncertainty  [22]  and  dimensionality  reduction. 

An  inherent  difficulty  resides  in  the  fusion  of  only  two 
forms  of  data  with  conflicts.  Sensor  conflicts  increase  the 
cognitive  workload  and  delay  action.  However,  if  time, 
space,  and  spectral  events  from  different  sensors  are 
fused  and  displayed,  conflicts  can  be  resolved  with  an 
emergent  process  to  allow  for  user  refinement  over 
varying  perspectives.  Displays  give  the  user  a  global  and 
local  SA/IA  to  guide  attention,  reduce  workload,  increase 
trust,  and  afford  action.  A  formal  Fusion  Usability 
Evaluation  (user-centered  or  user-driven)  is  needed  to 
define  process  interactions  for  users  to  complete  their 
tasks. 

5  Summary 

This  paper  provides  insights  into  user  information  needs 
from  situation/impact  assessment  for  sensor,  user,  and 
mission  (SUM)  management.  The  DFIG  supports  effective 
and  efficient  proactive  decision  making.  Information 
fusion  SUM  issues,  are  (1)  designing  for  users,  (2) 
determining  a  standard  set  of  metrics  for  cost  function 
optimization,  (3)  evaluating  systems  to  support  effective 
decision  making,  (4)  requiring  decentralized  updates  for 
mission  planning,  and  (5)  fusion  interface  guidelines  to 
support  user’s  control  actions. 
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